Uczenie Maszynowe - LAB2b - LIME¶

Biblioteka LIME: https://github.com/marcotcr/lime (Dokumentacja API: https://lime-ml.readthedocs.io/en/latest/)

Wprowadzenie - pakiety¶

Niezbędne pakiety i moduły na potrzeby wprowadzenia

InĀ [5]:
import json
from functools import partial

import matplotlib.pyplot as plt
import numpy as np
import torch
from lime import lime_image
from PIL import Image
from skimage.segmentation import mark_boundaries
from torchvision import models, transforms

Wprowadzenie - funkcje pomocnicze¶

Funkcja do wczytywania wskazanego obrazka oraz konwersji do palety RGB.

InĀ [6]:
def get_image(path):
    with open(path, 'rb') as f:
        with Image.open(f) as img:
            return img.convert('RGB')

Funkcja do przekształcania obrazka (zwróconego przez funkcję get_image) w tensor, akceptowalny na wejściu sieci neronowej.

InĀ [7]:
def image_to_tensor(img):    
    transformer = transforms.Compose([
        transforms.Resize((256, 256)),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(
            mean=[0.485, 0.456, 0.406],
            std=[0.229, 0.224, 0.225])
    ])  
    return transformer(img).unsqueeze(0)

Funkcja używana przez LIME, przyjmuje na wejściu zbiór obrazków, a zwraca prawdopodobieństwa klas. Należy ją przekazać do lime_image.LimeImageExplainer().explain_instance przy użyciu partial, jako partial(predict_batch, <model>), gdzie modelem w naszym wypadku będą sieci neuronowe. Przykłady użycia są zawarte w tym notebooku.

InĀ [8]:
def predict_batch(model, images):
    model.eval()
    transformer = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(
            mean=[0.485, 0.456, 0.406],
            std=[0.229, 0.224, 0.225]) 
    ])   
    
    model.eval()
    batch = torch.stack(tuple(transformer(i) for i in images), dim=0)

    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model.to(device)
    batch = batch.to(device)
    
    logits = model(batch)
    probas = torch.nn.functional.softmax(logits, dim=1)
    return probas.detach().cpu().numpy()

Funkcja, która przekształca obrazek w format akceptowany na wejściu przez LIME. Przykłady użycia są zawarte w tym notebooku.

InĀ [9]:
def lime_transformer(image):
    transformer = transforms.Compose([
        transforms.Resize((256, 256)),
        transforms.CenterCrop(224)
    ])
    return np.array(transformer(image))

LIME jest głównie wykorzystywane do wyjaśniania predykcji tzw. czarnych skrzynek, czyli modeli nieinterpretowalnych. Idealnymi kandydatami są Głębokie Sieci Neuronowe, dlatego spróbujemy wyjaśnić niektóre predykcje gotowych modeli.

Model Inception-v3 - przygotowanie danych¶

https://arxiv.org/abs/1512.00567

Plik ./data/imagenet_class_index.json zawiera przypisanie klas obrazków do indeksów. Jest to istotne, ponieważ zwracane wyniki (np. wartości funkcji logit na wyjściu sieci neuronowych) wykorzystują to, zwracając wyniki w zadanej kolejności.

InĀ [10]:
with open("./data/imagenet_class_index.json") as f:
    content = json.load(f)
    index_to_label = {
        int(index): data[1]
        for index, data in content.items()
    }
InĀ [11]:
image_to_classify = get_image("./data/dogs.png")
plt.imshow(image_to_classify)
Out[11]:
<matplotlib.image.AxesImage at 0x2428a17dcd0>
No description has been provided for this image
InĀ [12]:
img_tensor = image_to_tensor(image_to_classify)

Załadowanie pretrenowanego modelu¶

InĀ [13]:
inception_v3 = models.inception_v3(pretrained=True)
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=Inception_V3_Weights.IMAGENET1K_V1`. You can also use `weights=Inception_V3_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
Downloading: "https://download.pytorch.org/models/inception_v3_google-0cc3c7bd.pth" to C:\Users\Monika/.cache\torch\hub\checkpoints\inception_v3_google-0cc3c7bd.pth
100%|ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆ| 104M/104M [00:02<00:00, 54.2MB/s] 

Predykcja¶

InĀ [14]:
inception_v3.eval()
logits = inception_v3(img_tensor)

Zwróć uwagę, że model zwraca wartości funkcji logit, a nie prawdopodobieństwa klas, dlatego wyniki trzeba przetworzyć (np. przy użyciu funkcji softmax).

InĀ [17]:
probas = torch.nn.functional.softmax(logits, dim=1)

SprawdÅŗmy N najbardziej prawdopodobnych klas

InĀ [18]:
TOP_N_LABELS = 15

probas_top = probas.topk(TOP_N_LABELS)
top_probas = probas_top[0][0].detach().numpy()
top_labels = probas_top[1][0].detach().numpy()
for proba, label in zip(top_probas, top_labels):
    print(f"Class: {index_to_label[label]:<30} | Probability: {proba:.6f}")
Class: Bernese_mountain_dog           | Probability: 0.935930
Class: EntleBucher                    | Probability: 0.038448
Class: Appenzeller                    | Probability: 0.023756
Class: Greater_Swiss_Mountain_dog     | Probability: 0.001818
Class: Gordon_setter                  | Probability: 0.000009
Class: Blenheim_spaniel               | Probability: 0.000007
Class: English_springer               | Probability: 0.000002
Class: tabby                          | Probability: 0.000002
Class: robin                          | Probability: 0.000001
Class: guinea_pig                     | Probability: 0.000001
Class: amphibian                      | Probability: 0.000001
Class: Japanese_spaniel               | Probability: 0.000001
Class: African_grey                   | Probability: 0.000001
Class: Brittany_spaniel               | Probability: 0.000001
Class: toucan                         | Probability: 0.000001

Teraz możemy te funkcje zebrać razem¶

InĀ [19]:
def get_prediction_probabilities(image, model):
    img_tensor = image_to_tensor(image)
    model.eval()
    logits = model(img_tensor)
    probas = torch.nn.functional.softmax(logits, dim=1)
    
    TOP_N_LABELS = 15

    probas_top = probas.topk(TOP_N_LABELS)
    top_probas = probas_top[0][0].detach().numpy()
    top_labels = probas_top[1][0].detach().numpy()
    for proba, label in zip(top_probas, top_labels):
        print(f"Class: {index_to_label[label]:<30} | Probability: {proba:.6f}")

I sprawdzić jak ta predykcja wygląda dla innego obrazka¶

InĀ [20]:
exercise_image = get_image("./data/cat_mouse.jpeg")
plt.imshow(exercise_image)
Out[20]:
<matplotlib.image.AxesImage at 0x2428a6ca510>
No description has been provided for this image

Zadanie: sprawdÅŗ jak będzie wyglądała predykcja dla powyższego obrazka¶

InĀ [21]:
get_prediction_probabilities(exercise_image, inception_v3)
Class: Egyptian_cat                   | Probability: 0.967492
Class: tabby                          | Probability: 0.024167
Class: lynx                           | Probability: 0.005490
Class: tiger_cat                      | Probability: 0.002165
Class: Persian_cat                    | Probability: 0.000105
Class: Angora                         | Probability: 0.000074
Class: swab                           | Probability: 0.000071
Class: Madagascar_cat                 | Probability: 0.000064
Class: snow_leopard                   | Probability: 0.000040
Class: tile_roof                      | Probability: 0.000037
Class: indri                          | Probability: 0.000020
Class: leopard                        | Probability: 0.000016
Class: Siamese_cat                    | Probability: 0.000011
Class: ram                            | Probability: 0.000010
Class: crate                          | Probability: 0.000009

Model Inception-v3 - wyjaśnienie¶

Chcemy wiedzieć dlaczego klasa Bernese_mountain_dog została uznana przez sieć neuronową za najbardziej prawdopodobną (to znaczy - które piksele obrazka o tym zadecydowały). W tym celu właśnie wykorzystamy LIME.

W jaki sposób działa LIME na obrazkach?

  1. Na wejściu wymagany jest oryginalny obrazek.
  2. Wejściowy obrazek jest delikatnie przekształcany wiele razy, dzięki czemu otrzymujemy wiele podobnych (ale nie takich samych!) obrazków.
  3. Dodatkowo na wejście musimy podać funkcję, która każdemu takiemu przekształceniu nada prawdopodobieństwo przynależności do danej klasy. Jest to wymagane ponieważ LIME jest niezależny od żadnych narzędzi i modeli.
InĀ [22]:
explainer = lime_image.LimeImageExplainer()
InĀ [23]:
explanation = explainer.explain_instance(
    image=lime_transformer(image_to_classify), 
    classifier_fn=partial(predict_batch, inception_v3),
    top_labels=5,
    num_samples=1000)
  0%|          | 0/1000 [00:00<?, ?it/s]

Mając te dane możemy teraz sprawdzić które kategorie są najbardziej prawdopodobne

InĀ [24]:
for index in explanation.top_labels:
    print(index_to_label[index])
Bernese_mountain_dog
EntleBucher
Appenzeller
Greater_Swiss_Mountain_dog
Gordon_setter

Zobaczmy co wpłynęło na wybranie Bernese_mountain_dog jako najbardziej prawdopodobnej klasy.

InĀ [25]:
image, mask = explanation.get_image_and_mask(
    label=explanation.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[25]:
<matplotlib.image.AxesImage at 0x2428a6deed0>
No description has been provided for this image

Zadanie: zmień wartość NUM_FEATURES i zaobserwuj jak zmienia się mapowanie¶

NUM_FEATURES najlepiej zmieniać w zakresie 1:50

InĀ [26]:
NUM_FEATURES = 10

image, mask = explanation.get_image_and_mask(
    label=explanation.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=NUM_FEATURES,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[26]:
<matplotlib.image.AxesImage at 0x2428a769010>
No description has been provided for this image
InĀ [27]:
NUM_FEATURES = 30

image, mask = explanation.get_image_and_mask(
    label=explanation.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=NUM_FEATURES,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[27]:
<matplotlib.image.AxesImage at 0x2428a7b8110>
No description has been provided for this image
InĀ [28]:
NUM_FEATURES = 1

image, mask = explanation.get_image_and_mask(
    label=explanation.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=NUM_FEATURES,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[28]:
<matplotlib.image.AxesImage at 0x2428a6ca8a0>
No description has been provided for this image

The bigger NUM_FEATURES is the bigger area is marked. For too small value unimportant area is marked.

Zielone fragmenty oznaczają "superpiksele", które pozytywnie wpływają na predykowaną klasę. Czerwone fragmenty wpływają negatywnie.

Zadanie-pytanie: co to właściwie jest superpiksel?¶

Superpixel is an area which is the most important part of this object to classify to it's class.

Zadanie-pytanie: czy jeden superpiksel ma odzwierciedlenie w jednym pikselu z obrazka?¶

No, one superpixel is much bigger than just a pixel.

Zobaczmy jak to się prezentuje dla drugiej najbardziej prawdopodobnej klasy, czyli EntleBucher, która jednak otrzymała jedyne 3.8%.

InĀ [29]:
image, mask = explanation.get_image_and_mask(
    label=explanation.top_labels[1],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[29]:
<matplotlib.image.AxesImage at 0x2428fd0e360>
No description has been provided for this image

Ustawiając wartości hide_rest oraz positive_only na True jesteśmy w stanie zostawić tylko te piksele, które potwierdzały przynależność do danej klasy
Musimy jednak pamiętać o przeskalowaniu rezultatu przy pomocy (boundaries).astype(np.uint8)

InĀ [30]:
image, mask = explanation.get_image_and_mask(
    label=explanation.top_labels[0],
    positive_only=True,
    negative_only=False,
    num_features=10,
    hide_rest=True)
boundaries = mark_boundaries(image, mask)
plt.imshow((boundaries).astype(np.uint8))
Out[30]:
<matplotlib.image.AxesImage at 0x2428fcf3740>
No description has been provided for this image

Możemy również zostawić tylko te piksele, które zaprzeczały przynależności do danej klasy

InĀ [31]:
image, mask = explanation.get_image_and_mask(
    label=explanation.top_labels[0],
    positive_only=False,
    negative_only=True,
    num_features=10,
    hide_rest=True)
boundaries = mark_boundaries(image, mask)
cropped_image_ndarray = (boundaries).astype(np.uint8)
plt.imshow(cropped_image_ndarray)
Out[31]:
<matplotlib.image.AxesImage at 0x24290101c10>
No description has been provided for this image

A następnie sprawdzić co model sądzi o tak wyciętym obrazku

InĀ [32]:
cropped_image_pil = Image.fromarray(cropped_image_ndarray)

get_prediction_probabilities(cropped_image_pil, inception_v3)
Class: lumbermill                     | Probability: 0.272693
Class: triceratops                    | Probability: 0.241274
Class: pedestal                       | Probability: 0.118357
Class: sundial                        | Probability: 0.081880
Class: fountain                       | Probability: 0.029933
Class: caldron                        | Probability: 0.029304
Class: African_grey                   | Probability: 0.018543
Class: vulture                        | Probability: 0.017247
Class: stone_wall                     | Probability: 0.013481
Class: cannon                         | Probability: 0.011382
Class: black_grouse                   | Probability: 0.009726
Class: frilled_lizard                 | Probability: 0.009235
Class: chain_saw                      | Probability: 0.007655
Class: grand_piano                    | Probability: 0.007258
Class: conch                          | Probability: 0.006042

I jak go teraz widzi model

InĀ [33]:
cropped_image_explanation = explainer.explain_instance(
    image=lime_transformer(cropped_image_pil), 
    classifier_fn=partial(predict_batch, inception_v3),
    top_labels=5,
    num_samples=1000)
  0%|          | 0/1000 [00:00<?, ?it/s]
InĀ [34]:
image, mask = cropped_image_explanation.get_image_and_mask(
    label=cropped_image_explanation.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[34]:
<matplotlib.image.AxesImage at 0x2428a6c9880>
No description has been provided for this image

Model Inception-v3 - porównanie z AlexNet¶

Przetestujmy działanie na innym modelu - AlexNet

InĀ [35]:
alexnet = models.alexnet(pretrained=True)
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
Downloading: "https://download.pytorch.org/models/alexnet-owt-7be5be79.pth" to C:\Users\Monika/.cache\torch\hub\checkpoints\alexnet-owt-7be5be79.pth
100%|ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆ| 233M/233M [00:05<00:00, 47.6MB/s] 
InĀ [36]:
explanation_alexnet = explainer.explain_instance(
    image=lime_transformer(image_to_classify), 
    classifier_fn=partial(predict_batch, alexnet),
    top_labels=5,
    num_samples=1000)
  0%|          | 0/1000 [00:00<?, ?it/s]
InĀ [37]:
for index_alex, index_inception in zip(explanation_alexnet.top_labels, explanation.top_labels):
    print(f"{index_to_label[index_alex]:30} | {index_to_label[index_inception]:30}")
Bernese_mountain_dog           | Bernese_mountain_dog          
EntleBucher                    | EntleBucher                   
Greater_Swiss_Mountain_dog     | Appenzeller                   
Appenzeller                    | Greater_Swiss_Mountain_dog    
basset                         | Gordon_setter                 

Jak widać, klasy nieco się różnią, ale TOP 1 pozostaje takie samo.

InĀ [38]:
image, mask = explanation_alexnet.get_image_and_mask(
    label=explanation_alexnet.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[38]:
<matplotlib.image.AxesImage at 0x242905e2030>
No description has been provided for this image

Wyjaśnienie dla AlexNet jak się można było spodziewać - też się różni, jednak w dalszym ciągu do klasyfikacji psa istotny jest... pies :)

Zadanie: porównaj predykcje obrazka dla modeli inception_v3 oraz alexnet¶

InĀ [39]:
print("inception_v3")
get_prediction_probabilities(image_to_classify, inception_v3)
print()
print("alexnet")
get_prediction_probabilities(image_to_classify, alexnet)
inception_v3
Class: Bernese_mountain_dog           | Probability: 0.935930
Class: EntleBucher                    | Probability: 0.038448
Class: Appenzeller                    | Probability: 0.023756
Class: Greater_Swiss_Mountain_dog     | Probability: 0.001818
Class: Gordon_setter                  | Probability: 0.000009
Class: Blenheim_spaniel               | Probability: 0.000007
Class: English_springer               | Probability: 0.000002
Class: tabby                          | Probability: 0.000002
Class: robin                          | Probability: 0.000001
Class: guinea_pig                     | Probability: 0.000001
Class: amphibian                      | Probability: 0.000001
Class: Japanese_spaniel               | Probability: 0.000001
Class: African_grey                   | Probability: 0.000001
Class: Brittany_spaniel               | Probability: 0.000001
Class: toucan                         | Probability: 0.000001

alexnet
Class: Bernese_mountain_dog           | Probability: 0.400542
Class: EntleBucher                    | Probability: 0.208726
Class: Greater_Swiss_Mountain_dog     | Probability: 0.194558
Class: Appenzeller                    | Probability: 0.186891
Class: basset                         | Probability: 0.002400
Class: Cardigan                       | Probability: 0.002184
Class: Border_collie                  | Probability: 0.001509
Class: kelpie                         | Probability: 0.000398
Class: bluetick                       | Probability: 0.000314
Class: Blenheim_spaniel               | Probability: 0.000276
Class: collie                         | Probability: 0.000255
Class: borzoi                         | Probability: 0.000188
Class: Walker_hound                   | Probability: 0.000177
Class: Tibetan_mastiff                | Probability: 0.000171
Class: English_springer               | Probability: 0.000156

Zadanie domowe - wstęp¶

W folderze data znajduje się zdjęcie amfibii: title

InĀ [40]:
amphibious_vehicle = get_image("./data/amfibia.jpg")
InĀ [55]:
inception_v3 = models.inception_v3()

explanation_amhibious_vehicle_inception_v3 = explainer.explain_instance(
    image=lime_transformer(amphibious_vehicle), 
    classifier_fn=partial(predict_batch, inception_v3),
    top_labels=5,
    num_samples=1000)

image, mask = explanation_amhibious_vehicle_inception_v3.get_image_and_mask(
    label=explanation_amhibious_vehicle_inception_v3.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=Inception_V3_Weights.IMAGENET1K_V1`. You can also use `weights=Inception_V3_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
  0%|          | 0/1000 [00:00<?, ?it/s]
Out[55]:
<matplotlib.image.AxesImage at 0x2428a299640>
No description has been provided for this image

Model inception_v3 jak i jego wyjaśnienie rzeczywiście sugerują amfibię jako najbardziej prawdopodobną klasę:

InĀ [56]:
for index in explanation_amhibious_vehicle_inception_v3.top_labels:
    print(index_to_label[index])
amphibian
convertible
racer
car_wheel
golfcart

Zadanie #1¶

Użyj dwóch różnych sieci neuronowych (poza inception_v3, którego przykład jest powyżej) do wygenerowania wyjaśnień.
(skorzystaj z modułu torchvision: https://pytorch.org/vision/stable/models.html)

alexNet¶

InĀ [53]:
alexnet = models.alexnet()

explanation_amhibious_vehicle_alexnet = explainer.explain_instance(
    image=lime_transformer(amphibious_vehicle), 
    classifier_fn=partial(predict_batch, alexnet),
    top_labels=5,
    num_samples=1000)

image, mask = explanation_amhibious_vehicle_alexnet.get_image_and_mask(
    label=explanation_amhibious_vehicle_alexnet.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
  0%|          | 0/1000 [00:00<?, ?it/s]
Out[53]:
<matplotlib.image.AxesImage at 0x24296462ff0>
No description has been provided for this image
InĀ [54]:
for index in explanation_amhibious_vehicle_alexnet.top_labels:
    print(index_to_label[index])
ashcan
great_grey_owl
pickup
cardigan
weevil

GoogleNet¶

InĀ [45]:
googlenet = models.googlenet(pretrained=True)

explanation_amhibious_vehicle_googlenet = explainer.explain_instance(
    image=lime_transformer(amphibious_vehicle), 
    classifier_fn=partial(predict_batch, googlenet),
    top_labels=5,
    num_samples=1000)

image, mask = explanation_amhibious_vehicle_googlenet.get_image_and_mask(
    label=explanation_amhibious_vehicle_googlenet.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=GoogLeNet_Weights.IMAGENET1K_V1`. You can also use `weights=GoogLeNet_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
Downloading: "https://download.pytorch.org/models/googlenet-1378be20.pth" to C:\Users\Monika/.cache\torch\hub\checkpoints\googlenet-1378be20.pth
100%|ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆ| 49.7M/49.7M [00:01<00:00, 50.1MB/s]
  0%|          | 0/1000 [00:00<?, ?it/s]
Out[45]:
<matplotlib.image.AxesImage at 0x2428a29bef0>
No description has been provided for this image
InĀ [46]:
for index in explanation_amhibious_vehicle_googlenet.top_labels:
    print(index_to_label[index])
pickup
tow_truck
jeep
amphibian
convertible

Zadanie #2¶

Zmodyfikuj oryginalny obrazek w taki sposób, żeby najbardziej prawdopodobną klasą dla każdej z tych sieci nie była amfibia a jakiś inny pojazd (np. samochód). W tym celu możesz "zasłonić" czarnym kwadratem (wartość 0 w macierzy reprezentującej obraz) obszary istotne przy klasyfikacji.
Przydatną rzeczą będzie skorzystanie z opcji hide_rest w funkcji get_image_and_mask i późniejsza obróbka obrazu

InĀ [86]:
# Only the inception_v3 has the most probably amphibia
inception_v3 = models.inception_v3()

explanation_amhibious_vehicle_inception_v3 = explainer.explain_instance(
    image=lime_transformer(amphibious_vehicle), 
    classifier_fn=partial(predict_batch, inception_v3),
    top_labels=5,
    num_samples=1000)
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\inception.py:43: FutureWarning: The default weight initialization of inception_v3 will be changed in future releases of torchvision. If you wish to keep the old behavior (which leads to long initialization times due to scipy/scipy#11299), please set init_weights=True.
  warnings.warn(
  0%|          | 0/1000 [00:00<?, ?it/s]
InĀ [109]:
image, mask = explanation_amhibious_vehicle_inception_v3.get_image_and_mask(
    label=explanation_amhibious_vehicle_inception_v3.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[109]:
<matplotlib.image.AxesImage at 0x24298989e20>
No description has been provided for this image
InĀ [114]:
modified_image = image.copy()
positive_mask = mask.astype(bool)
modified_image[positive_mask] = [0, 0, 0]
plt.imshow(modified_image)
plt.axis('off')
plt.show()
No description has been provided for this image
InĀ [119]:
modified_image = Image.fromarray(modified_image)
InĀ [120]:
explanation_amhibious_vehicle_inception_v3_modified = explainer.explain_instance(
    image=lime_transformer(modified_image), 
    classifier_fn=partial(predict_batch, inception_v3),
    top_labels=5,
    num_samples=1000)
  0%|          | 0/1000 [00:00<?, ?it/s]
InĀ [121]:
for index in explanation_amhibious_vehicle_inception_v3_modified.top_labels:
    print(index_to_label[index])
cairn
bulbul
coral_fungus
tench
toilet_tissue

Zadanie #3¶

Ponownie zmodyfikuj oryginalny obraz, ale tym razem zaszumiając go w losowy sposób (przykładowa implementacja: https://www.geeksforgeeks.org/add-a-salt-and-pepper-noise-to-an-image-with-python/). Czy najbardziej prawdopodobna klasa zmienia się wraz ze zmianą szumu? Przetestuj dla każdego z modeli.

InĀ [122]:
import random 
import cv2 

def add_noise(img): 
    row , col = img.shape 
    # white
    number_of_pixels = random.randint(300, 10000) 
    for i in range(number_of_pixels): 
        y_coord=random.randint(0, row - 1) 
        x_coord=random.randint(0, col - 1) 
        img[y_coord][x_coord] = 255
          
    # black
    number_of_pixels = random.randint(300 , 10000) 
    for i in range(number_of_pixels): 
        y_coord=random.randint(0, row - 1) 
        x_coord=random.randint(0, col - 1) 
        img[y_coord][x_coord] = 0
          
    return img 
  
img = cv2.imread('./data/amfibia.jpg', 
                 cv2.IMREAD_GRAYSCALE) 

cv2.imwrite('salt-and-pepper-amfibia.jpg', 
            add_noise(img)) 
Out[122]:
True
InĀ [123]:
amphibious_noise = get_image('salt-and-pepper-amfibia.jpg')
plt.imshow(amphibious_noise)
plt.axis('off')
plt.show()
No description has been provided for this image

Inception_v3¶

InĀ [125]:
inception_v3 = models.inception_v3()

explanation_amhibious_vehicle_inception_v3 = explainer.explain_instance(
    image=lime_transformer(amphibious_noise), 
    classifier_fn=partial(predict_batch, inception_v3),
    top_labels=5,
    num_samples=1000)
  0%|          | 0/1000 [00:00<?, ?it/s]
InĀ [133]:
image, mask = explanation_amhibious_vehicle_inception_v3.get_image_and_mask(
    label=explanation_amhibious_vehicle_inception_v3.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
plt.axis('off')
plt.show()
No description has been provided for this image
InĀ [127]:
for index in explanation_amhibious_vehicle_inception_v3.top_labels:
    print(index_to_label[index])
Cardigan
bulbul
coral_fungus
tench
toilet_tissue

AlexNet¶

InĀ [134]:
alexnet = models.alexnet()

explanation_amhibious_vehicle_alexnet = explainer.explain_instance(
    image=lime_transformer(amphibious_noise), 
    classifier_fn=partial(predict_batch, alexnet),
    top_labels=5,
    num_samples=1000)

image, mask = explanation_amhibious_vehicle_alexnet.get_image_and_mask(
    label=explanation_amhibious_vehicle_alexnet.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
plt.axis('off')
plt.show()
  0%|          | 0/1000 [00:00<?, ?it/s]
No description has been provided for this image
InĀ [129]:
for index in explanation_amhibious_vehicle_alexnet.top_labels:
    print(index_to_label[index])
brambling
cricket
English_setter
triumphal_arch
head_cabbage

GoogLeNet¶

InĀ [130]:
googlenet = models.googlenet(pretrained=True)

explanation_amhibious_vehicle_googlenet = explainer.explain_instance(
    image=lime_transformer(amphibious_noise), 
    classifier_fn=partial(predict_batch, googlenet),
    top_labels=5,
    num_samples=1000)

image, mask = explanation_amhibious_vehicle_googlenet.get_image_and_mask(
    label=explanation_amhibious_vehicle_googlenet.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=GoogLeNet_Weights.IMAGENET1K_V1`. You can also use `weights=GoogLeNet_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
  0%|          | 0/1000 [00:00<?, ?it/s]
Out[130]:
<matplotlib.image.AxesImage at 0x2429e77ee40>
No description has been provided for this image
InĀ [131]:
for index in explanation_amhibious_vehicle_googlenet.top_labels:
    print(index_to_label[index])
amphibian
jeep
tow_truck
snowplow
half_track

Using various models the most probably class is changing. Only GoogLeNet still is finding amphibian as first choice and futher other car types.